Capturing and Processing of Images Captured by Non-Grid Camera Arrays

ABSTRACT

A camera array, an imaging device and/or a method for capturing image that employ a plurality of imagers fabricated on a substrate is provided. Each imager includes a plurality of pixels. The plurality of imagers include a first imager having a first imaging characteristics and a second imager having a second imaging characteristics. The images generated by the plurality of imagers are processed to obtain an enhanced image compared to images captured by the imagers. Each imager may be associated with an optical element fabricated using a wafer level optics (WLO) technology.

RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/447,477, entitled “Systems and Methods for Parallax Measurement UsingCamera Arrays Incorporating 3×3 Camera Configurations”, filed on Jul.30, 2014, which application is a continuation of U.S. patent applicationSer. No. 12/935,504, entitled “Capturing and Processing of Images UsingMonolithic Camera Array with Heterogeneous Imagers”, filed on Sep. 29,2010, which application was a 35 U.S.C. 371 national stage applicationcorresponding to Application No. PCT/US2009/044687 filed May 20, 2009,which claims priority to U.S. Provisional Patent Application No.61/054,694 entitled “Monolithic Integrated Array of Heterogeneous ImageSensors,” filed on May 20, 2008, which is incorporated by referenceherein in its entirety.

FIELD OF THE INVENTION

The present invention is related to an image sensor including aplurality of heterogeneous imagers, more specifically to an image sensorwith a plurality of wafer-level imagers having custom filters, sensorsand optics of varying configurations.

BACKGROUND

Image sensors are used in cameras and other imaging devices to captureimages. In a typical imaging device, light enters through an opening(aperture) at one end of the imaging device and is directed to an imagesensor by an optical element such as a lens. In most imaging devices,one or more layers of optical elements are placed between the apertureand the image sensor to focus light onto the image sensor. The imagesensor consists of pixels that generate signals upon receiving light viathe optical element. Commonly used image sensors include CCD(charge-coupled device) image sensors and CMOS (complementarymetal-oxide-semiconductor) sensors.

Filters are often employed in the image sensor to selectively transmitlights of certain wavelengths onto pixels. A Bayer filter mosaic isoften formed on the image sensor. The Bayer filter is a color filterarray that arranges one of the RGB color filters on each of the colorpixels. The Bayer filter pattern includes 50% green filters, 25% redfilters and 25% blue filters. Since each pixel generates a signalrepresenting strength of a color component in the light and not the fullrange of colors, demosaicing is performed to interpolate a set of red,green and blue values for each image pixel.

The image sensors are subject to various performance constraints. Theperformance constraints for the image sensors include, among others,dynamic range, signal to noise (SNR) ratio and low light sensitivity.The dynamic range is defined as the ratio of the maximum possible signalthat can be captured by a pixel to the total noise signal. Typically,the well capacity of an image sensor limits the maximum possible signalthat can be captured by the image sensor. The maximum possible signal inturn is dependent on the strength of the incident illumination and theduration of exposure (e.g., integration time, and shutter width). Thedynamic range can be expressed as a dimensionless quantity in decibels(dB) as:

$\begin{matrix}{{DR} = \frac{{full}\mspace{14mu} {well}\mspace{14mu} {capacity}}{{RMS}\mspace{14mu} {noise}}} & {{equation}\mspace{14mu} (1)}\end{matrix}$

Typically, the noise level in the captured image influences the floor ofthe dynamic range. Thus, for an 8 bit image, the best case would be 48dB assuming the RMS noise level is 1 bit. In reality, however, the RMSnoise levels are higher than 1 bit, and this further reduces the dynamicrange.

The signal to noise ratio (SNR) of a captured image is, to a greatextent, a measure of image quality. In general, as more light iscaptured by the pixel, the higher the SNR. The SNR of a captured imageis usually related to the light gathering capability of the pixel.

Generally, Bayer filter sensors have low light sensitivity. At low lightlevels, each pixel's light gathering capability is constrained by thelow signal levels incident upon each pixel. In addition, the colorfilters over the pixel further constrain the signal reaching the pixel.IR (Infrared) filters also reduce the photo-response from near-IRsignals, which can carry valuable information.

These performance constraints of image sensors are greatly magnified incameras designed for mobile systems due to the nature of designconstraints. Pixels for mobile cameras are typically much smaller thanthe pixels of digital still cameras (DSC). Due to limits in lightgathering ability, reduced SNR, limits in the dynamic range, and reducedsensitivity to low light scenes, the cameras in mobile cameras show poorperformance.

SUMMARY

Embodiments provide a camera array, an imaging device including a cameraarray and/or a method for capturing image that employ a plurality ofimagers fabricated on a substrate where each imager includes a pluralityof sensor elements. The plurality of imagers include at least a firstimager formed on a first location of the substrate and a second imagerformed on a second location of the substrate. The first imager and thesecond imager may have the same imaging characteristics or differentimaging characteristics.

In one embodiment, the first imaging characteristics and the secondimager have different imaging characteristics. The imagingcharacteristics may include, among others, the size of the imager, thetype of pixels included in the imager, the shape of the imager, filtersassociated with the imager, exposure time of the imager, aperture sizeassociated with the imager, the configuration of the optical elementassociated with the imager, gain of the imager, the resolution of theimager, and operational timing of the imager.

In one embodiment, the first imager includes a filter for transmitting alight spectrum. The second imager also includes the same type of filterfor transmitting the same light spectrum as the first imager butcaptures an image that is sub-pixel phase shifted from an image capturedby the first imager. The images from the first imager and the secondimager are combined using a super-resolution process to obtain images ofhigher resolution.

In one embodiment, the first imager includes a first filter fortransmitting a first light spectrum and the second imager includes asecond filter for transmitting a second light spectrum. The images fromthe first and second imagers are then processed to obtain a higherquality image.

In one embodiment, lens elements are provided to direct and focus lightonto the imagers. Each lens element focuses light onto one imager.Because each lens element is associated with one imager, each lenselement may be designed and configured for a narrow light spectrum.Further, the thickness of the lens element may be reduced, decreasingthe overall thickness of the camera array. The lens elements arefabricated using wafer level optics (WLO) technology.

In one embodiment, the plurality of imagers include at least one near-IRimager dedicated to receiving near-IR (Infrared) spectrum. An imagegenerated from the near-IR imager may be fused with images generatedfrom other imagers with color filters to reduce noise and increase thequality of the images.

In one embodiment, the plurality of imagers may be associated with lenselements that provide a zooming capability. Different imagers may beassociated with lens of different focal lengths to have differentfields-of-views and provide different levels of zooming capability. Amechanism may be provided to provide smooth transition from one zoomlevel to another zoom level.

In one or more embodiments, the plurality of imagers is coordinated andoperated to obtain at least one of a high dynamic range image, apanoramic image, a hyper-spectral image, distance to an object and ahigh frame rate video.

The features and advantages described in the specification are not allinclusive and, in particular, many additional features and advantageswill be apparent to one of ordinary skill in the art in view of thedrawings, specification, and claims. Moreover, it should be noted thatthe language used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the inventive subject matter.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a plan view of a camera array with a plurality of imagers,according to one embodiment.

FIG. 2A is a perspective view of a camera array with lens elements,according to one embodiment.

FIG. 2B is a cross-sectional view of a camera array, according to oneembodiment.

FIGS. 3A and 3B are sectional diagrams illustrating changes in theheights of lens elements depending on changes in the dimensions ofimagers, according to one embodiment.

FIG. 3C is a diagram illustrating chief ray angles varying depending ondiffering dimensions of the lens elements.

FIG. 4 is a functional block diagram for an imaging device, according toone embodiment.

FIG. 5 is a functional block diagram of an image processing pipelinemodule, according to one embodiment.

FIGS. 6A through 6E are plan views of camera arrays having differentlayouts of heterogeneous imagers, according to embodiments.

FIG. 7 is a flowchart illustrating a process of generating an enhancedimage from lower resolution images captured by a plurality of imagers,according to one embodiment.

DETAILED DESCRIPTION

A preferred embodiment of the present invention is now described withreference to the figures where like reference numbers indicate identicalor functionally similar elements. Also in the figures, the left mostdigits of each reference number corresponds to the figure in which thereference number is first used.

Embodiments relate to using a distributed approach to capturing imagesusing a plurality of imagers of different imaging characteristics. Eachimager may be spatially shifted from another imager in such a mannerthat an imager captures an image that us shifted by a sub-pixel amountwith respect to another imager captured by another imager. Each imagermay also include separate optics with different filters and operate withdifferent operating parameters (e.g., exposure time). Distinct imagesgenerated by the imagers are processed to obtain an enhanced image. Eachimager may be associated with an optical element fabricated using waferlevel optics (WLO) technology.

A sensor element or pixel refers to an individual light sensing elementin a camera array. The sensor element or pixel includes, among others,traditional CIS (CMOS Image Sensor), CCD (charge-coupled device), highdynamic range pixel, multispectral pixel and various alternativesthereof.

An imager refers to a two dimensional array of pixels. The sensorelements of each imager have similar physical properties and receivelight through the same optical component. Further, the sensor elementsin the each imager may be associated with the same color filter.

A camera array refers to a collection of imagers designed to function asa unitary component. The camera array may be fabricated on a single chipfor mounting or installing in various devices.

An array of camera array refers to an aggregation of two or more cameraarrays. Two or more camera arrays may operate in conjunction to provideextended functionality over a single camera array.

Image characteristics of an imager refer to any characteristics orparameters of the imager associated with capturing of images. Theimaging characteristics may include, among others, the size of theimager, the type of pixels included in the imager, the shape of theimager, filters associated with the imager, the exposure time of theimager, aperture size associated with the imager, the configuration ofthe optical element associated with the imager, gain of the imager, theresolution of the imager, and operational timing of the imager.

Structure of Camera Array

FIG. 1 is a plan view of a camera array 100 with imagers 1A through NM,according to one embodiment. The camera array 100 is fabricated on asemiconductor chip to include a plurality of imagers 1A through NM. Eachof the imagers 1A through NM may include a plurality of pixels (e.g.,0.32 Mega pixels). In one embodiment, the imagers 1A through NM arearranged into a grid format as illustrated in FIG. 1. In otherembodiments, the imagers are arranged in a non-grid format. For example,the imagers may be arranged in a circular pattern, zigzagged pattern orscattered pattern.

The camera array may include two or more types of heterogeneous imagers,each imager including two or more sensor elements or pixels. Each one ofthe imagers may have different imaging characteristics. Alternatively,there may be two or more different types of imagers where the same typeof imagers shares the same imaging characteristics.

In one embodiment, each imager 1A through NM has its own filter and/oroptical element (e.g., lens). Specifically, each of the imagers 1Athrough NM or a group of imagers may be associated with spectral colorfilters to receive certain wavelengths of light. Example filters includea traditional filter used in the Bayer pattern (R, G, B or theircomplements C, M, Y), an IR-cut filter, a near-IR filter, a polarizingfilter, and a custom filter to suit the needs of hyper-spectral imaging.Some imagers may have no filter to allow reception of both the entirevisible spectra and near-IR, which increases the imager'ssignal-to-noise ratio. The number of distinct filters may be as large asthe number of imagers in the camera array. Further, each of the imagers1A through NM or a group of imagers may receive light through lenshaving different optical characteristics (e.g., focal lengths) orapertures of different sizes.

In one embodiment, the camera array includes other related circuitry.The other circuitry may include, among others, circuitry to controlimaging parameters and sensors to sense physical parameters. The controlcircuitry may control imaging parameters such as exposure times, gain,and black level offset. The sensor may include dark pixels to estimatedark current at the operating temperature. The dark current may bemeasured for on-the-fly compensation for any thermal creep that thesubstrate may suffer from.

In one embodiment, the circuit for controlling imaging parameters maytrigger each imager independently or in a synchronized manner. The startof the exposure periods for the various imagers in the camera array(analogous to opening a shutter) may be staggered in an overlappingmanner so that the scenes are sampled sequentially while having severalimagers being exposed to light at the same time. In a conventional videocamera sampling a scene at N exposures per second, the exposure time persample is limited to 1/N seconds. With a plurality of imagers, there isno such limit to the exposure time per sample because multiple imagersmay be operated to capture images in a staggered manner.

Each imager can be operated independently. Entire or most operationsassociated with each individual imager may be individualized. In oneembodiment, a master setting is programmed and deviation (i.e., offsetor gain) from such master setting is configured for each imager. Thedeviations may reflect functions such as high dynamic range, gainsettings, integration time settings, digital processing settings orcombinations thereof. These deviations can be specified at a low level(e.g., deviation in the gain) or at a higher level (e.g., difference inthe ISO number, which is then automatically translated to deltas forgain, integration time, or otherwise as specified by context/mastercontrol registers) for the particular camera array. By setting themaster values and deviations from the master values, higher levels ofcontrol abstraction can be achieved to facilitate simpler programmingmodel for many operations. In one embodiment, the parameters for theimagers are arbitrarily fixed for a target application. In anotherembodiment, the parameters are configured to allow a high degree offlexibility and programmability.

In one embodiment, the camera array is designed as a drop-in replacementfor existing camera image sensors used in cell phones and other mobiledevices. For this purpose, the camera array may be designed to bephysically compatible with conventional image sensors of approximatelythe same resolution although the achieved resolution of the camera arraymay exceed conventional image sensors in many photographic situations.Taking advantage of the increased performance, the camera array of theembodiment may include fewer pixels to obtain equal or better qualityimages compared to conventional image sensors. Alternatively, the sizeof the pixels in the imager may be reduced compared to pixels inconventional image sensors while achieving comparable results.

In order to match the raw pixel count of a conventional image sensorwithout increasing silicon area, the logic overhead for the individualimagers is preferably constrained in the silicon area. In oneembodiment, much of the pixel control logic is a single collection offunctions common to all or most of the imagers with a smaller set offunctions applicable each imager. In this embodiment, the conventionalexternal interface for the imager may be used because the data outputdoes not increase significantly for the imagers.

In one embodiment, the camera array including the imagers replaces aconventional image sensor of M megapixels. The camera array includes N×Nimagers, each sensor including pixels of

$\frac{M}{N^{2}}.$

Each imager in the camera array also has the same aspect ratio as theconventional image sensor being replaced. Table 1 lists exampleconfigurations of camera arrays according to the present inventionreplacing conventional image sensor.

TABLE 1 Conventional Camera array Including Imagers Image Sensor No. ofNo. of Total Effective Total Horizontal Vertical Imager Super-ResolutionEffective Mpixels Resolution Mpixels Imagers Imagers Mpixels FactorResolution 8 3.2 8 5 5 0.32 3.2 3.2 8 4 4 0.50 2.6 3.2 8 3 3 0.89 1.93.2 5 2.0 5 5 5 0.20 3.2 2.0 5 4 4 0.31 2.6 2.0 5 3 3 0.56 1.9 2.0 3 1.23 5 5 0.12 3.2 1.2 3 4 4 0.19 2.6 1.2 3 3 3 0.33 1.9 1.2

The Super-Resolution Factors in Table 1 are estimates and the EffectiveResolution values may differ based on the actual Super-Resolutionfactors achieved by processing.

The number of imagers in the camera array may be determined based on,among other factors, (i) resolution, (ii) parallax, (iii) sensitivity,and (iv) dynamic range. A first factor for the size of imager is theresolution. From a resolution point of view, the preferred number of theimagers ranges from 2×2 to 6×6 because an array size of larger than 6×6is likely to destroy frequency information that cannot be recreated bythe super-resolution process. For example, 8 Megapixel resolution with2×2 imager will require each imager to have 2 Megapixels. Similarly, 8Megapixel resolution with a 5×5 array will require each imager to have0.32 Megapixels.

A second factor that may constrain the number of imagers is the issue ofparallax and occlusion. With respect to an object captured in an image,the portion of the background scene that is occluded from the view ofthe imager is called as “occlusion set.” When two imagers capture theobject from two different locations, the occlusion set of each imager isdifferent. Hence, there may be scene pixels captured by one imager butnot the other. To resolve this issue of occlusion, it is desirable toinclude a certain minimal set of imagers for a given type of imager.

A third factor that may put a lower bound on the number of imagers isthe issue of sensitivity in low light conditions. To improve low lightsensitivity, imagers for detecting near-IR spectrum may be needed. Thenumber of imagers in the camera array may need to be increased toaccommodate such near-IR imagers.

A fourth factor in determining the size of the imager is dynamic range.To provide dynamic range in the camera array, it is advantageous toprovide several imagers of the same filter type (chroma or luma). Eachimager of the same filter type may then be operated with differentexposures simultaneously. The images captured with different exposuresmay be processed to generate a high dynamic range image.

Based on these factors, the preferred number of imagers is 2×2 to 6×6.4×4 and 5×5 configurations are more preferable than 2×2 and 3×3configurations because the former are likely to provide sufficientnumber of imagers to resolve occlusion issues, increase sensitivity andincrease the dynamic range. At the same time, the computational loadrequired to recover resolution from these array sizes will be modest incomparison to that required in the 6×6 array. Arrays larger than 6×6may, however, be used to provide additional features such as opticalzooming and multispectral imaging.

Another consideration is the number of imagers dedicated to lumasampling. By ensuring that the imagers in the array dedicated to near-IRsampling do not reduce the achieved resolution, the information from thenear-IR images is added to the resolution captured by the luma imagers.For this purpose, at least 50% of the imagers may be used for samplingthe luma and/or near-IR spectra. In one embodiment with 4×4 imagers, 4imagers samples luma, 4 imagers samples near-IR, and the remaining 8imagers samples two chroma (Red and Blue). In another embodiment with5×5 imagers, 9 imagers samples luma, 8 imagers samples near-IR, and theremaining 8 imagers samples two chroma (Red and Blue). Further, theimagers with these filters may be arranged symmetrically within thecamera array to address occlusion due to parallax.

In one embodiment, the imagers in the camera array are spatiallyseparated from each other by a predetermined distance. By increasing thespatial separation, the parallax between the images captured by theimagers may be increased. The increased parallax is advantageous wheremore accurate distance information is important. Separation between twoimagers may also be increased to approximate the separation of a pair ofhuman eyes. By approximating the separation of human eyes, a realisticstereoscopic 3D image may be provided to present the resulting image onan appropriate 3D display device.

In one embodiment, multiple camera arrays are provided at differentlocations on a device to overcome space constraints. One camera arraymay be designed to fit within a restricted space while another cameraarray may be placed in another restricted space of the device. Forexample, if a total of 20 imagers are required but the available spaceallows only a camera array of 1×10 imagers to be provided on either sideof a device, two camera arrays each including 10 imagers may be placedon available space at both sides of the device. Each camera array may befabricated on a substrate and be secured to a motherboard or other partsof a device. The images collected from multiple camera arrays may beprocessed to generate images of desired resolution and performance.

A design for a single imager may be applied to different camera arrayseach including other types of imagers. Other variables in the cameraarray such as spatial distances, color filters and combination with thesame or other sensors may be modified to produce a camera array withdiffering imaging characteristics. In this way, a diverse mix of cameraarrays may be produced while maintaining the benefits from economies ofscale.

Wafer Level Optics Integration

In one embodiment, the camera array employs wafer level optics (WLO)technology. WLO is a technology that molds optics on glass wafersfollowed by packaging of the optics directly with the imager into amonolithic integrated module. The WLO procedure may involve, among otherprocedures, using a diamond-turned mold to create each plastic lenselement on a glass substrate.

FIG. 2A is a perspective view of a camera array assembly 200 with waferlevel optics 210 and a camera array 230, according to one embodiment.The wafer level optics 210 includes a plurality of lens elements 220,each lens element 220 covering one of twenty-five imagers 240 in thecamera array 230. Note that the camera array assembly 200 has an arrayof smaller lens elements occupy much less space compared to a singlelarge lens covering the entire camera array 230.

FIG. 2B is a sectional view of a camera array assembly 250, according toone embodiment. The camera assembly 250 includes a top lens wafer 262, abottom lens wafer 268, a substrate 278 with multiple imagers formedthereon and spacers 258, 264 270. The camera array assembly 250 ispackaged within an encapsulation 254. A top spacer 258 is placed betweenthe encapsulation 254 and the top lens wafer 262. Multiple opticalelements 288 are formed on the top lens wafer 262. A middle spacer 264is placed between the top lens wafer 262 and a bottom lens wafer 268.Another set of optical elements 286 is formed on the bottom lens wafer268. A bottom spacer 270 is placed between the bottom lens wafer 268 andthe substrate 278. Through-silicon vias 274 are also provided to pathsfor transmitting signal from the imagers. The top lens wafer 262 may bepartially coated with light blocking materials 284 (e.g., chromium) toblock of light. The portions of the top lens wafer 262 not coated withthe blocking materials 284 serve as apertures through which light passesto the bottom lens wafer 268 and the imagers. In the embodiment of FIG.2B, filters 282 are formed on the bottom lens wafer 268. Light blockingmaterials 280 (e.g., chromium) may also be coated on the bottom lens 268and the substrate 278 to function as an optical isolator. The bottomsurface of the surface is covered with a backside redistribution layer(“RDL”) and solder balls 276.

In one embodiment, the camera array assembly 250 includes 5×5 array ofimagers. The camera array 250 has a width W of 7.2 mm, and a length of8.6 mm. Each imager in the camera array may have a width S of 1.4 mm.The total height t1 of the optical components is approximately 1.26 mmand the total height t2 the camera array assembly is less than 2 mm.

FIGS. 3A and 3B are diagrams illustrating changes in the height t of alens element pursuant to changes in dimensions in an x-y plane. A lenselement 320 in FIG. 3B is scaled by 1/n compared to a lens element 310in FIG. 3A. As the diameter L/n of the lens element 320 is smaller thanthe diameter L by a factor of n, the height t/n of the lens element 320is also smaller than the height t of the lens element 310 by a factor ofn. Hence, by using an array of smaller lens elements, the height of thecamera array assembly can be reduced significantly. The reduced heightof the camera array assembly may be used to design less aggressivelenses having better optical properties such as improved chief rayangle, reduced distortion, and improved color aberration.

FIG. 3C illustrates improving a chief ray angle (CRA) by reducing thethickness of the camera array assembly. CRA1 is the chief ray angle fora single lens covering an entire camera array. Although the chief rayangle can be reduced by increasing the distance between the camera arrayand the lens, the thickness constraints imposes constraints onincreasing the distance. Hence, the CRA1 for camera array having asingle lens element is large, resulting in reduced optical performance.CRA2 is the chief ray angle for an imager in the camera array that isscaled in thickness as well as other dimensions. The CRA2 remains thesame as the CRA1 of the conventional camera array and results in noimprovement in the chief ray angle. By modifying the distance betweenthe imager and the lens element as illustrated in FIG. 3C, however, thechief ray angle CRA3 in the camera array assembly may be reducedcompared to CRA1 or CRA2, resulting in better optical performance. Asdescribed above, the camera arrays according to the present inventionhas reduced thickness requirements, and therefore, the distance of thelens element and the camera array may be increased to improve the chiefray angle.

In addition, the lens elements are subject to less rigorous designconstraints yet produces better or equivalent performance compared toconventional lens element covering a wide light spectrum because eachlens element may be designed to direct a narrow band of light. Forexample, an imager receiving visible or near-IR spectrum may have a lenselement specifically optimized for this spectral band of light. Forimagers detecting other light spectrum, the lens element may havediffering focal lengths so that the focal plane is the same fordifferent spectral bands of light. The matching of the focal planeacross different wavelengths of light increases the sharpness of imagecaptured at the imager and reduces longitudinal chromatic aberration.

Other advantages of smaller lens element include, among others, reducedcost, reduced amount of materials, and the reduction in themanufacturing steps. By providing n²lenses that are 1/n the size in xand y dimension (and thus 1/n thickness), the wafer size for producingthe lens element may also be reduced. This reduces the cost and theamount of materials considerably. Further, the number of lens substrateis reduced, which results in reduced number of manufacturing steps andreduced attendant yield costs. The placement accuracy required toregister the lens array to the imagers is typically no more stringentthan in the case of a conventional imager because the pixel size for thecamera array according to the present invention may be substantiallysame as a conventional image sensor

In one embodiment, the WLO fabrication process includes: (i)incorporating lens element stops by plating the lens element stops ontothe substrate before lens molding, and (ii) etching holes in thesubstrate and performing two-sided molding of lenses through thesubstrate. The etching of holes in the substrate is advantageous becauseindex mismatch is not caused between plastic and substrate. In this way,light absorbing substrate that forms natural stops for all lens elements(similar to painting lens edges black) may be used.

In one embodiment, filters are part of the imager. In anotherembodiment, filters are part of a WLO subsystem.

Imaging System and Processing Pipeline

FIG. 4 is a functional block diagram illustrating an imaging system 400,according to one embodiment. The imaging system 400 may include, amongother components, the camera array 410, an image processing pipelinemodule 420 and a controller 440. The camera array 410 includes two ormore imagers, as described above in detail with reference to FIGS. 1 and2. Images 412 are captured by the two or more imagers in the cameraarray 410.

The controller 440 is hardware, software, firmware or a combinationthereof for controlling various operation parameters of the camera array410. The controller 440 receives inputs 446 from a user or otherexternal components and sends operation signals 442 to control thecamera array 410. The controller 440 may also send information 444 tothe image processing pipeline module 420 to assist processing of theimages 412.

The image processing pipeline module 420 is hardware, firmware, softwareor a combination for processing the images received from the cameraarray 410. The image processing pipeline module 420 processes multipleimages 412, for example, as described below in detail with reference toFIG. 5. The processed image 422 is then sent for display, storage,transmittal or further processing.

FIG. 5 is a functional block diagram illustrating the image processingpipeline module 420, according to one embodiment. The image processingpipeline module 420 may include, among other components, an upstreampipeline processing module 510, an image pixel correlation module 514, aparallax confirmation and measurement module 518, a parallaxcompensation module 522, a super-resolution module 526, an addressconversion module 530, an address and phase offset calibration module554, and a downstream color processing module 564.

The address and phase offset calibration module 554 is a storage devicefor storing calibration data produced during camera arraycharacterization in the manufacturing process or a subsequentrecalibration process. The calibration data indicates mapping betweenthe addresses of physical pixels 572 in the imagers and the logicaladdresses 546, 548 of an image.

The address conversion module 530 performs normalization based on thecalibration data stored in the address and phase offset calibrationmodule 554. Specifically, the address conversion module 530 converts“physical” addresses of the individual pixels in the image to “logical”addresses 548 of the individual pixels in the imagers or vice versa. Inorder for super-resolution processing to produce an image of enhancedresolution, the phase difference between corresponding pixels in theindividual imagers needs to be resolved. The super-resolution processmay assume that for each pixel in the resulting image the set of inputpixels from each of the imager is consistently mapped and that the phaseoffset for each imager is already known with respect to the position ofthe pixel in the resulting image. The address conversion module 530resolves such phase differences by converting the physical addresses inthe images 412 into logical addresses 548 of the resulting image forsubsequent processing.

The images 412 captured by the imagers 540 are provided to the upstreampipeline processing module 510. The upstream pipe processing module 510may perform one or more of Black Level calculation and adjustments,fixed noise compensation, optical PSF (point spread function)deconvolution, noise reduction, and crosstalk reduction. After the imageis processed by the upstream pipeline processing module 510, an imagepixel correlation module 514 performs calculation to account forparallax that becomes more apparent as objects being captured approachesto the camera array. Specifically, the image pixel correlation module514 aligns portions of images captured by different imagers tocompensate for the parallax. In one embodiment, the image pixelcorrelation module 514 compares the difference between the averagevalues of neighboring pixels with a threshold and flags the potentialpresence of parallax when the difference exceeds the threshold. Thethreshold may change dynamically as a function of the operatingconditions of the camera array. Further, the neighborhood calculationsmay also be adaptive and reflect the particular operating conditions ofthe selected imagers.

The image is then processed by the parallax confirmation and measurementmodule 518 to detect and meter the parallax. In one embodiment, parallaxdetection is accomplished by a running pixel correlation monitor. Thisoperation takes place in logical pixel space across the imagers withsimilar integration time conditions. When the scene is at practicalinfinity, the data from the imagers is highly correlated and subjectonly to noise-based variations. When an object is close enough to thecamera, however, a parallax effect is introduced that changes thecorrelation between the imagers. Due to the spatial layout of theimagers, the nature of the parallax-induced change is consistent acrossall imagers. Within the limits of the measurement accuracy, thecorrelation difference between any pair of imagers dictates thedifference between any other pair of imagers and the differences acrossthe other imagers. This redundancy of information enables highlyaccurate parallax confirmation and measurement by performing the same orsimilar calculations on other pairs of imagers. If parallax is presentin the other pairs, the parallax should occur at roughly the samephysical location of the scene taking into account the positions of theimagers. The measurement of the parallax may be accomplished at the sametime by keeping track of the various pair-wise measurements andcalculating an “actual” parallax difference as a least squares (orsimilar statistic) fit to the sample data. Other methods for detectingthe parallax may include detecting and tracking vertical and horizontalhigh-frequency image elements from frame-to-frame.

The parallax compensation module 522 processes images including objectsclose enough to the camera array to induce parallax differences largerthan the accuracy of the phase offset information required by superresolution process. The parallax compensation module 522 uses thescan-line based parallax information generated in the parallax detectionand measurement module 518 to further adjust mapping between physicalpixel addresses and logical pixel addresses before the super-resolutionprocess. There are two cases that occur during this processing. In amore common case, addressing and offsetting adjustment are required whenthe input pixels have shifted positions relative to theimage-wise-corresponding pixels in other imagers. In this case, nofurther processing with respect to parallax is required beforeperforming super-resolution. In a less common case, a pixel or group ofpixels are shifted in such a way that exposes the occlusion set. In thiscase, the parallax compensation process generates tagged pixel dataindicating that the pixels of the occlusion set should not be consideredin the super-resolution process.

After the parallax change has been accurately determined for aparticular imager, the parallax information 524 is sent to the addressconversion module 530. The address conversion module 530 uses theparallax information 524 along with the calibration data 558 from theaddress and phase offset calibration module 554 to determine theappropriate X and Y offsets to be applied to logical pixel addresscalculations. The address conversion module 530 also determines theassociated sub-pixel offset for a particular imager pixel with respectto pixels in the resulting image 428 produced by the super-resolutionprocess. The address conversion module 530 takes into account theparallax information 524 and provides logical addresses 546 accountingfor the parallax.

After performing the parallax compensation, the image is processed bythe super-resolution module 526 to obtain a high resolution synthesizedimage 422 from low resolution images, as described below in detail. Thesynthesized image 422 may then be fed to the downstream color processingmodule 564 to perform one or more of the following operations: focusrecover, white balance, color correction, gamma correction, RGB to YUVcorrection, edge-aware sharpening, contrast enhancement and compression.

The image processing pipeline module 420 may include components foradditional processing of the image. For example, the image processingpipeline module 420 may include a correction module for correctingabnormalities in images caused by a single pixel defect or a cluster ofpixel defects. The correction module may be embodied on the same chip asthe camera array, as a component separate from the camera array or as apart of the super-resolution module 526.

Super-Resolution Processing

In one embodiment, the super-resolution module 526 generates a higherresolution synthesized image by processing low resolution imagescaptured by the imagers 540. The overall image quality of thesynthesized image is higher than images captured from any one of theimagers individually. In other words, the individual imagers operatesynergistically, each contributing to higher quality images using theirability to capture a narrow part of the spectrum without sub-sampling.The image formation associated with the super-resolution techniques maybe expressed as follows:

y _(k) =W _(k) ·x+n _(k) , ∀k=1 . . . p   equation (2)

where W_(k) represents the contribution of the HR scene (x) (viablurring, motion, and sub-sampling) to each of the LR images (y_(k))captured on each of the k imagers and n_(k) is the noise contribution.

FIGS. 6A through 6E illustrate various configurations of imagers forobtaining a high resolution image through a super-resolution process,according to embodiments of the present invention. In FIGS. 6A through4E, “R” represents an imager having a red filter, “G” represents aimager having a green filter, “B” represents an imager having a bluefilter, “P” represents a polychromatic imager having sensitivity acrossthe entire visible spectra and near-IR spectrum, and “I” represents animager having a near-IR filter. The polychromatic imager may sampleimage from all parts of the visible spectra and the near-IR region(i.e., from 650 nm to 800 nm). In the embodiment of FIG. 6A, the centercolumns and rows of the imagers include polychromatic imagers. Theremaining areas of the camera array are filled with imagers having greenfilters, blue filters, and red filters. The embodiment of FIG. 6A doesnot include any imagers for detecting near-IR spectrum alone.

The embodiment of FIG. 6B has a configuration similar to conventionalBayer filter mapping. This embodiment does not include any polychromaticimagers or near-IR imagers. As described above in detail with referenceto FIG. 1, the embodiment of FIG. 6B is different from conventionalBayer filter configuration in that each color filter is mapped to eachimager instead of being mapped to an individual pixel.

FIG. 6C illustrates an embodiment where the polychromatic imagers form asymmetric checkerboard pattern. FIG. 6D illustrates an embodiment wherefour near-IR imagers are provided. FIG. 6E illustrates an embodimentwith irregular mapping of imagers. The embodiments of FIGS. 6A through6E are merely illustrative and various other layouts of imagers can alsobe used.

The use of polychromatic imagers and near-IR imagers is advantageousbecause these sensors may capture high quality images in low lightingconditions. The images captured by the polychromatic imager or thenear-IR imager are used to denoise the images obtained from regularcolor imagers.

The premise of increasing resolution by aggregating multiple lowresolution images is based on the fact that the different low resolutionimages represent slightly different viewpoints of the same scene. If theLR images are all shifted by integer units of a pixel, then each imagecontains essentially the same information. Therefore, there is no newinformation in LR images that can be used to create the HR image. In theimagers according to embodiments, the layout of the imagers may bepreset and controlled so that each imager in a row or a column is afixed sub-pixel distance from its neighboring imagers. The wafer levelmanufacturing and packaging process allows accurate formation of imagersto attain the sub-pixel precisions required for the super-resolutionprocessing.

An issue of separating the spectral sensing elements into differentimagers is parallax caused by the physical separation of the imagers. Byensuring that the imagers are symmetrically placed, at least two imagerscan capture the pixels around the edge of a foreground object. In thisway, the pixels around the edge of a foreground object may be aggregatedto increase resolution as well as avoiding any occlusions. Another issuerelated to parallax is the sampling of color. The issue of sampling thecolor may be reduced by using parallax information in the polychromaticimagers to improve the accuracy of the sampling of color from the colorfiltered imagers.

In one embodiment, near-IR imagers are used to determine relativeluminance differences compared to a visible spectra imager. Objects havediffering material reflectivity results in differences in the imagescaptured by the visible spectra and the near-IR spectra. At low lightingconditions, the near-IR imager exhibits a higher signal to noise ratios.Therefore, the signals from the near-IR sensor may be used to enhancethe luminance image. The transferring of details from the near-IR imageto the luminance image may be performed before aggregating spectralimages from different imagers through the super-resolution process. Inthis way, edge information about the scene may be improved to constructedge-preserving images that can be used effectively in thesuper-resolution process. The advantage of using near-IR imagers isapparent from equation (2) where any improvement in the estimate for thenoise (i.e., n) leads to a better estimate of the original HR scene (x).

FIG. 7 is a flowchart illustrating a process of generating an HR imagefrom LR images captured by a plurality of imagers, according to oneembodiment. First, luma images, near-IR images and chroma images arecaptured 710 by imagers in the camera array. Then normalization isperformed 714 on the captured images to map physical addresses of theimagers to logical addresses in the enhanced image. Parallaxcompensation is then performed 720 to resolve any differences in thefield-of-views of the imagers due to spatial separations between theimagers. Super-resolution processing is then performed 724 to obtainsuper-resolved luma images, super-resolved near-IR images, andsuper-resolved chroma images.

Then it is determined 728 if the lighting condition is better than apreset parameter. If the lighting condition is better than theparameter, the process proceeds to normalize 730 a super-resolvednear-IR image with respect to a super-resolved luma image. A focusrecovery is then performed 742. In one embodiment, the focus recovery isperformed 742 using PSF (point spread function) deblurring per eachchannel. Then the super-resolution is processed 746 based on near-IRimages and the luma images. A synthesized image is then constructed 750.

If it is determined 728 that the lighting condition is not better thanthe preset parameter, the super-resolved near-IR images and luma imagesare aligned 734. Then the super-resolved luma images are denoised 738using the near-IR super-resolved images. Then the process proceeds toperforming focus recovery 742 and repeats the same process as when thelighting condition is better than the preset parameter. Then the processterminates.

Image Fusion of Color Images with Near-IR Images

The spectral response of CMOS imagers is typically very good in thenear-IR regions covering 650 nm to 800 nm and reasonably good between800 nm and 1000 nm. Although near-IR images having no chromainformation, information in this spectral region is useful in lowlighting conditions because the near-IR images are relatively free ofnoise. Hence, the near-IR images may be used to denoise color imagesunder the low lighting conditions.

In one embodiment, an image from a near-IR imager is fused with anotherimage from a visible light imager. Before proceeding with the fusion, aregistration is performed between the near-IR image and the visiblelight image to resolve differences in viewpoints. The registrationprocess may be performed in an offline, one-time, processing step. Afterthe registration is performed, the luminance information on the near-IRimage is interpolated to grid points that correspond to each grid pointon the visible light image.

After the pixel correspondence between the near-IR image and the visiblelight image is established, denoising and detail transfer process may beperformed. The denoising process allows transfer of signal informationfrom the near-IR image to the visible light image to improve the overallSNR of the fusion image. The detail transfer ensures that edges in thenear-IR image and the visible light image are preserved and accentuatedto improve the overall visibility of objects in the fused image.

In one embodiment, a near-IR flash may serve as a near-IR light sourceduring capturing of an image by the near-IR imagers. Using the near-IRflash is advantageous, among other reasons, because (i) the harshlighting on objects of interest may be prevented, (ii) ambient color ofthe object may be preserved, and (iii) red-eye effect may be prevented.

In one embodiment, a visible light filter that allows only near-IR raysto pass through is used to further optimize the optics for near-IRimaging. The visible light filter improves the near-IR optics transferfunction because the light filter results in sharper details in thenear-IR image. The details may then be transferred to the visible lightimages using a dual bilateral filter as described, for example, in EricP. Bennett et al., “Multispectral Video Fusion,” Computer Graphics (ACMSIGGRAPH Proceedings) (Jul. 25, 2006), which is incorporated byreference herein in its entirety.

Dynamic Range Determination by Differing Exposures at Imagers

An auto-exposure (AE) algorithm is important to obtaining an appropriateexposure for the scene to be captured. The design of the AE algorithmaffects the dynamic range of captured images. The AE algorithmdetermines an exposure value that allows the acquired image to fall inthe linear region of the camera array's sensitivity range. The linearregion is preferred because a good signal-to-noise ratio is obtained inthis region. If the exposure is too low, the picture becomesunder-saturated while if the exposure is too high the picture becomesover-saturated. In conventional cameras, an iterative process is takento reduce the difference between measured picture brightness andpreviously defined brightness below a threshold. This iterative processrequires a large amount of time for convergence, and sometimes resultsin an unacceptable shutter delay.

In one embodiment, the picture brightness of images captured by aplurality of imagers is independently measured. Specifically, aplurality of imagers are set to capturing images with differentexposures to reduce the time for computing the adequate exposure. Forexample, in a camera array with 5×5 imagers where 8 luma imagers and 9near-IR imagers are provided, each of the imagers may be set withdifferent exposures. The near-IR imagers are used to capture low-lightaspects of the scene and the luma imagers are used to capture the highillumination aspects of the scene. This results in a total of 17possible exposures. If exposure for each imager is offset from anadjacent imager by a factor of 2, for example, a maximum dynamic rangeof 2¹⁷ or 102 dB can be captured. This maximum dynamic range isconsiderably higher than the typical 48 dB attainable in a conventionalcamera with 8 bit image outputs.

At each time instant, the responses (under-exposed, over-exposed oroptimal) from each of the multiple imagers are analyzed based on howmany exposures are needed at the subsequent time instant. The ability toquery multiple exposures simultaneously in the range of possibleexposures accelerates the search compared to the case where only oneexposure is tested at once. By reducing the processing time fordetermining the adequate exposure, shutter delays and shot-to-shot lagsmay be reduced.

In one embodiment, the HDR image is synthesized from multiple exposuresby combining the images after linearizing the imager response for eachexposure. The images from the imagers may be registered before combiningto account for the difference in the viewpoints of the imagers.

In one embodiment, at least one imager includes HDR pixels to generateHDR images. HDR pixels are specialized pixels that capture high dynamicrange scenes. Although HDR pixels show superior performances compared toother pixels, HDR pixels show poor performance at low lightingconditions in comparison with near-IR imagers. To improve performance atlow lighting conditions, signals from the near-IR imagers may be used inconjunction with the signal from the HDR imager to attain better qualityimages across different lighting conditions.

In one embodiment, an HDR image is obtained by processing imagescaptured by multiple imagers by processing, as disclosed, for example,in Paul Debevec et al., “Recovering High Dynamic Range Radiance Mapsfrom Photographs,” Computer Graphics (ACM SIGGRAPH Proceedings), (Aug.16, 1997), which is incorporated by reference herein in its entirety.The ability to capture multiple exposures simultaneously using theimager is advantageous because artifacts caused by motion of objects inthe scene can be mitigated or eliminated.

Hyperspectral Imaging by Multiple Imagers

In one embodiment, a multi-spectral image is rendered by multipleimagers to facilitate the segmentation or recognition of objects in ascene. Because the spectral reflectance coefficients vary smoothly inmost real world objects, the spectral reflectance coefficients may beestimated by capturing the scene in multiple spectral dimensions usingimagers with different color filters and analyzing the captured imagesusing Principal Components Analysis (PCA).

In one embodiment, half of the imagers in the camera array are devotedto sampling in the basic spectral dimensions (R, G, and B) and the otherhalf of the imagers are devoted to sampling in a shifted basic spectraldimensions (R′, G′, and B′). The shifted basic spectral dimensions areshifted from the basic spectral dimensions by a certain wavelength(e.g., 10 nm).

In one embodiment, pixel correspondence and non-linear interpolation isperformed to account for the sub-pixel shifted views of the scene. Thenthe spectral reflectance coefficients of the scene are synthesized usinga set of orthogonal spectral basis functions as disclosed, for example,in J. P. S. Parkkinen, J. Hallikainen and T. Jaaskelainen,“Characteristic Spectra of Munsell Colors,” J. Opt. Soc. Am., A 6:318(August 1989), which is incorporated by reference herein in itsentirety. The basis functions are eigenvectors derived by PCA of acorrelation matrix and the correlation matrix is derived from a databasestoring spectral reflectance coefficients measured by, for example,Munsell color chips (a total of 1257) representing the spectraldistribution of a wide range of real world materials to reconstruct thespectrum at each point in the scene.

At first glance, capturing different spectral images of the scenethrough different imagers in the camera array appears to traderesolution for higher dimensional spectral sampling. However, some ofthe lost resolution may be recovered. The multiple imagers sample thescene over different spectral dimensions where each sampling grid ofeach imager is offset by a sub-pixel shift from the others. In oneembodiment, no two sampling grid of the imager overlap. That is, thesuperposition of all the sampling grids from all the imagers forms adense, possibly non-uniform, montage of points. Scattered datainterpolation methods may be used to determine the spectral density ateach sample point in this non-uniform montage for each spectral image,as described, for example, in Shiaofen Fang et al., “Volume MorphingMethods for Landmark Based 3D Image Deformation” by SPIE vol. 2710,proc. 1996 SPIE Intl Symposium on Medical Imaging, page 404-415, NewportBeach, Calif. (February 1996), which is incorporated by reference hereinin its entirety. In this way, a certain amount of resolution lost in theprocess of sampling the scene using different spectral filters may berecovered.

As described above, image segmentation and object recognition arefacilitated by determining the spectral reflectance coefficients of theobject. The situation often arises in security applications wherein anetwork of cameras is used to track an object as it moves from theoperational zone of one camera to another. Each zone may have its ownunique lighting conditions (fluorescent, incandescent, D65, etc.) thatmay cause the object to have a different appearance in each imagecaptured by different cameras. If these cameras capture the images in ahyper-spectral mode, all images may be converted to the same illuminantto enhance object recognition performance.

In one embodiment, camera arrays with multiple imagers are used forproviding medical diagnostic images. Full spectral digitized images ofdiagnostic samples contribute to accurate diagnosis because doctors andmedical personnel can place higher confidence in the resultingdiagnosis. The imagers in the camera arrays may be provided with colorfilters to provide full spectral data. Such camera array may beinstalled on cell phones to capture and transmit diagnostic informationto remote locations as described, for example, in Andres W. Martinez etal., “Simple Telemedicine for Developing Regions: Camera Phones andPaper-Based Microfluidic Devices for Real-Time, Off-Site Diagnosis,”Analytical Chemistry (American Chemical Society) (Apr. 11, 2008), whichis incorporated by reference herein in its entirety. Further, the cameraarrays including multiple imagers may provide images with a large depthof field to enhance the reliability of image capture of wounds, rashes,and other symptoms.

In one embodiment, a small imager (including, for example, 20-500pixels) with a narrow spectral bandpass filters is used to produce asignature of the ambient and local light sources in a scene. By usingthe small imager, the exposure and white balance characteristics may bedetermined more accurately at a faster speed. The spectral bandpassfilters may be ordinary color filters or diffractive elements of abandpass width adequate to allow the number of camera arrays to coverthe visible spectrum of about 400 nm. These imagers may run at a muchhigher frame rate and obtain data (which may or may not be used for itspictorial content) for processing into information to control theexposure and white balance of other larger imagers in the same cameraarray. The small imagers may also be interspersed within the cameraarray.

Optical Zoom Implemented Using Multiple Imagers

In one embodiment, a subset of imagers in the camera array includestelephoto lenses. The subset of imagers may have other imagingcharacteristics same as imagers with non-telephoto lenses. Images fromthis subset of imagers are combined and super-resolution processed toform a super-resolution telephoto image. In another embodiment, thecamera array includes two or more subsets of imagers equipped withlenses of more than two magnifications to provide differing zoommagnifications.

Embodiments of the camera arrays may achieve its final resolution byaggregating images through super-resolution. Taking an example ofproviding 5×5 imagers with a 3× optical zoom feature, if 17 imagers areused to sample the luma (G) and 8 imagers are used to sample the chroma(R and B), 17 luma imagers allow a resolution that is four times higherthan what is achieved by any single imager in the set of 17 imagers. Ifthe number of the imager is increased from 5×5 to 6×6, an addition of 11extra imagers becomes available. In comparison with the 8 Megapixelconventional image sensor fitted with a 3× zoom lens, a resolution thatis 60% of the conventional image sensor is achieved when 8 of theadditional 11 imagers are dedicated to sampling luma (G) and theremaining 3 imagers are dedicated to chroma (R and B) and near-IRsampling at 3× zoom. This considerably reduces the chroma sampling (ornear-IR sampling) to luma sampling ratio. The reduced chroma to lumasampling ratio is somewhat offset by using the super-resolved luma imageat 3× zoom as a recognition prior on the chroma (and near-IR) image toresample the chroma image at a higher resolution.

With 6×6 imagers, a resolution equivalent to the resolution ofconventional image sensor is achieved at 1× zoom. At 3× zoom, aresolution equivalent to about 60% of conventional image sensoroutfitted with a 3× zoom lens is obtained by the same imagers. Also,there is a decrease in luma resolution at 3× zoom compared withconventional image sensors with resolution at 3× zoom. The decreasedluma resolution, however, is offset by the fact that the optics ofconventional image sensor has reduced efficiency at 3∴ zoom due tocrosstalk and optical aberrations.

The zoom operation achieved by multiple imagers has the followingadvantages. First, the quality of the achieved zoom is considerablyhigher than what is achieved in the conventional image sensor due to thefact that the lens elements may be tailored for each change in focallength. In conventional image sensors, optical aberrations and fieldcurvature must be corrected across the whole operating range of thelens, which is considerably harder in a zoom lens with moving elementsthan in a fixed lens element where only aberrations for a fixed focallength need to be corrected. Additionally, the fixed lens in the imagershas a fixed chief ray angle for a given height, which is not the casewith conventional image sensor with a moving zoom lens. Second, theimagers allow simulation of zoom lenses without significantly increasingthe optical track height. The reduced height allows implementation ofthin modules even for camera arrays with zooming capability.

The overhead required to support a certain level of optical zoom incamera arrays according to some embodiments is tabulated in Table 2.

TABLE 2 No. of No. of Luma No. of Chroma Imagers Imagers at Imagers atin different different Camera Zoom levels Zoom Levels array 1X 2X 3X 1X2X 3X 25 17 0 0 8 0 0 36 16 0 8 8 0 4

In one embodiment, the pixels in the images are mapped onto an outputimage with a size and resolution corresponding to the amount of zoomdesired in order to provide a smooth zoom capability from thewidest-angle view to the greatest-magnification view. Assuming that thehigher magnification lenses have the same center of view as the lowermagnification lenses, the image information available is such that acenter area of the image has a higher resolution available than theouter area. In the case of three or more distinct magnifications, nestedregions of different resolution may be provided with resolutionincreasing toward the center.

An image with the most telephoto effect has a resolution determined bythe super-resolution ability of the imagers equipped with the telephotolenses. An image with the widest field of view can be formatted in atleast one of two following ways. First, the wide field image may beformatted as an image with a uniform resolution where the resolution isdetermined by the super-resolution capability of the set of imagershaving the wider-angle lenses. Second, the wide field image is formattedas a higher resolution image where the resolution of the central part ofthe image is determined by the super-resolution capability of the set ofimagers equipped with telephoto lenses. In the lower resolution regions,information from the reduced number of pixels per image area isinterpolated smoothly across the larger number of “digital” pixels. Insuch an image, the pixel information may be processed and interpolatedso that the transition from higher to lower resolution regions occurssmoothly.

In one embodiment, zooming is achieved by inducing a barrel-likedistortion into some, or all, of the array lens so that adisproportionate number of the pixels are dedicated to the central partof each image. In this embodiment, every image has to be processed toremove the barrel distortion. To generate a wide angle image, pixelscloser to the center are sub-sampled relative to outer pixels aresuper-sampled. As zooming is performed, the pixels at the periphery ofthe imagers are progressively discarded and the sampling of the pixelsnearer the center of the imager is increased.

In one embodiment, mipmap filters are built to allow images to berendered at a zoom scale that is between the specific zoom range of theoptical elements (e.g., 1× and 3× zoom scales of the camera array).Mipmaps are a precalculated optimized set of images that accompany abaseline image. A set of images associated with the 3× zoom luma imagecan be created from a baseline scale at 3× down to 1×. Each image inthis set is a version of the baseline 3X zoom image but at a reducedlevel of detail. Rendering an image at a desired zoom level is achievedusing the mipmap by (i) taking the image at 1× zoom, and computing thecoverage of the scene for the desired zoom level (i.e., what pixels inthe baseline image needs to be rendered at the requested scale toproduce the output image), (ii) for each pixel in the coverage set,determine if the pixel is in the image covered by the 3× zoom lumaimage, (iii) if the pixel is available in the 3× zoom luma image, thenchoose the two closest mipmap images and interpolate (using smoothingfilter) the corresponding pixels from the two mipmap images to producethe output image, and (iv) if the pixel is unavailable in the 3× zoomluma image, then choose the pixel from the baseline 1× luma image andscale up to the desired scale to produce the output pixel. By usingmipmaps, smooth optical zoom may be simulated at any point between twogiven discrete levels (i.e., 1× zoom and 3× zoom).

Capturing Video Images

In one embodiment, the camera array generates high frame imagesequences. The imagers in the camera array can operate independently tocapture images. Compared to conventional image sensors, the camera arraymay capture images at the frame rate up to N time (where N is the numberof imagers). Further, the frame period for each imager may overlap toimprove operations under low-light conditions. To increase theresolution, a subset of imagers may operate in a synchronized manner toproduce images of higher resolution. In this case, the maximum framerate is reduced by the number of imagers operating in a synchronizedmanner. The high-speed video frame rates can enables slow-motion videoplayback at a normal video rate.

In one example, two luma imagers (green imagers or near-IR imagers), twoblue imagers and two green imagers are used to obtain high-definition1080p images. Using permutations of four luma imagers (two green imagersand two near-IR imagers or three green imagers and one near-IR imager)together with one blue imager and one red imager, the chroma imagers canbe upsampled to achieve 120 frames/sec for 1080p video. For higher framerate imaging devices, the number of frame rates can be scaled uplinearly. For Standard-Definition (480p) operation, a frame rate of 240frames/sec may be achieved using the same camera array.

Conventional imaging devices with a high-resolution image sensor (e.g.,8 Megapixels) use binning or skipping to capture lower resolution images(e.g., 1080p30, 720p30 and 480p30). In binning, rows and columns in thecaptured images are interpolated in the charge, voltage or pixel domainsin order to achieve the target video resolutions while reducing thenoise. In skipping, rows and columns are skipped in order to reduce thepower consumption of the sensor. Both of these techniques result inreduced image quality.

In one embodiment, the imagers in the camera arrays are selectivelyactivated to capture a video image. For example, 9 imagers (includingone near-IR imager) may be used to obtain 1080p (1920×1080 pixels)images while 6 imagers (including one near-IR imager) may be used toobtain 720p (1280×720 pixels) images or 4 imagers (including one near-IRimager) may be used to obtain 480p (720×480 pixels) images. Becausethere is an accurate one-to-one pixel correspondence between the imagerand the target video images, the resolution achieved is higher thantraditional approaches. Further, since only a subset of the imagers isactivated to capture the images, significant power savings can also beachieved. For example, 60% reduction in power consumption is achieved in1080p and 80% of power consumption is achieved in 480p.

Using the near-IR imager to capture video images is advantageous becausethe information from the near-IR imager may be used to denoise eachvideo image. In this way, the camera arrays of embodiments exhibitexcellent low-light sensitivity and can operate in extremely low-lightconditions. In one embodiment, super-resolution processing is performedon images from multiple imagers to obtain higher resolution videoimagers. The noise-reduction characteristics of the super-resolutionprocess along with fusion of images from the near-IR imager results in avery low-noise images.

In one embodiment, high-dynamic-range (HDR) video capture is enabled byactivating more imagers. For example, in a 5×5 camera array operating in1080p video capture mode, there are only 9 cameras active. A subset ofthe 16 cameras may be overexposed and underexposed by a stop in sets oftwo or four to achieve a video output with a very high dynamic range.

Other Applications for Multiple Imagers

In one embodiment, the multiple imagers are used for estimating distanceto an object in a scene. Since information regarding the distance toeach point in an image is available in the camera array along with theextent in x and y coordinates of an image element, the size of an imageelement may be determined. Further, the absolute size and shape ofphysical items may be measured without other reference information. Forexample, a picture of a foot can be taken and the resulting informationmay be used to accurately estimate the size of an appropriate shoe.

In one embodiment, reduction in depth of field is simulated in imagescaptured by the camera array using distance information. The cameraarrays according to the present invention produce images with greatlyincreased depth of field. The long depth of field, however, may not bedesirable in some applications. In such case, a particular distance orseveral distances may be selected as the “in best focus” distance(s) forthe image and based on the distance (z) information from parallaxinformation, the image can be blurred pixel-by-pixel using, for example,a simple Gaussian blur. In one embodiment, the depth map obtained fromthe camera array is utilized to enable a tone mapping algorithm toperform the mapping using the depth information to guide the level,thereby emphasizing or exaggerating the 3D effect.

In one embodiment, apertures of different sizes are provided to obtainaperture diversity. The aperture size has a direct relationship with thedepth of field. In miniature cameras, however, the aperture is generallymade as large as possible to allow as much light to reach the cameraarray. Different imagers may receive light through apertures ofdifferent sizes. For imagers to produce a large depth of field, theaperture may be reduced whereas other imagers may have large aperturesto maximize the light received. By fusing the images from sensor imagesof different aperture sizes, images with large depth of field may beobtained without sacrificing the quality of the image.

In one embodiment, the camera array according to the present inventionrefocuses based on images captured from offsets in viewpoints. Unlike aconventional plenoptic camera, the images obtained from the camera arrayof the present invention do not suffer from the extreme loss ofresolution. The camera array according to the present invention,however, produces sparse data points for refocusing compared to theplenoptic camera. In order to overcome the sparse data points,interpolation may be performed to refocus data from the spare datapoints.

In one embodiment, each imager in the camera array has a differentcentroid. That is, the optics of each imager are designed and arrangedso that the fields of view for each imager slightly overlap but for themost part constitute distinct tiles of a larger field of view. Theimages from each of the tiles are panoramically stitched together torender a single high-resolution image.

In one embodiment, camera arrays may be formed on separate substratesand mounted on the same motherboard with spatial separation. The lenselements on each imager may be arranged so that the corner of the fieldof view slightly encompasses a line perpendicular to the substrate.Thus, if four imagers are mounted on the motherboard with each imagerrotated 90 degrees with respect to another imager, the fields of viewwill be four slightly overlapping tiles. This allows a single design ofWLO lens array and imager chip to be used to capture different tiles ofa panoramic image.

In one embodiment, one or more sets of imagers are arranged to captureimages that are stitched to produce panoramic images with overlappingfields of view while another imager or sets of imagers have a field ofview that encompasses the tiled image generated. This embodimentprovides different effective resolution for imagers with differentcharacteristics. For example, it may be desirable to have more luminanceresolution than chrominance resolution. Hence, several sets of imagersmay detect luminance with their fields of view panoramically stitched.Fewer imagers may be used to detect chrominance with the field of viewencompassing the stitched field of view of the luminance imagers.

In one embodiment, the camera array with multiple imagers is mounted ona flexible motherboard such that the motherboard can be manually bent tochange the aspect ratio of the image. For example, a set of imagers canbe mounted in a horizontal line on a flexible motherboard so that in thequiescent state of the motherboard, the fields of view of all of theimagers are approximately the same. If there are four imagers, an imagewith double the resolution of each individual imager is obtained so thatdetails in the subject image that are half the dimension of details thatcan be resolved by an individual imager. If the motherboard is bent sothat it forms part of a vertical cylinder, the imagers point outward.With a partial bend, the width of the subject image is doubled while thedetail that can be resolved is reduced because each point in the subjectimage is in the field of view of two rather than four imagers. At themaximum bend, the subject image is four times wider while the detailthat can be resolved in the subject is further reduced.

Offline Reconstruction and Processing

The images processed by the imaging system 400 may be previewed beforeor concurrently with saving of the image data on a storage medium suchas a flash device or a hard disk. In one embodiment, the images or videodata includes rich light field data sets and other useful imageinformation that were originally captured by the camera array. Othertraditional file formats could also be used. The stored images or videomay be played back or transmitted to other devices over various wired orwireless communication methods.

In one embodiment, tools are provided for users by a remote server. Theremote server may function both as a repository and an offlineprocessing engine for the images or video. Additionally, applets mashedas part of popular photo-sharing communities such as Flikr, Picasaweb,Facebook etc. may allow images to be manipulated interactively, eitherindividually or collaboratively. Further, software plug-ins into imageediting programs may be provided to process images generated by theimaging device 400 on computing devices such as desktops and laptops.

Various modules described herein may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, application specific integratedcircuits (ASICs), or any type of media suitable for storing electronicinstructions, and each coupled to a computer system bus. Furthermore,the computers referred to in the specification may include a singleprocessor or may be architectures employing multiple processor designsfor increased computing capability.

While particular embodiments and applications of the present inventionhave been illustrated and described herein, it is to be understood thatthe invention is not limited to the precise construction and componentsdisclosed herein and that various modifications, changes, and variationsmay be made in the arrangement, operation, and details of the methodsand apparatuses of the present invention without departing from thespirit and scope of the invention as it is defined in the appendedclaims.

What is claimed is:
 1. A non-grid camera array, comprising: a pluralityof cameras arranged in a non-grid format and configured to captureimages of a scene, where each camera comprises: optics comprising atleast one lens element and at least one aperture; and a sensorcomprising a two dimensional array of pixels and control circuitry forcontrolling imaging parameters; a controller configured to controloperation parameters of the plurality of cameras; and an imageprocessing pipeline module configured to process at least a subset ofimages captured by the plurality of cameras; wherein the plurality ofcameras comprises at least two cameras having different imagingcharacteristics including different resolutions; wherein the imageprocessing pipeline module comprises a parallax confirmation andmeasurement module configured to: measure parallax within the processedimages by detecting parallax-induced changes that are consistent acrossthe images taking into account the position of the cameras that capturedthe images; and generate a depth map using the measured parallax.
 2. Thecamera array of claim 1, wherein the plurality of cameras comprises anarray of camera arrays.
 3. The camera array of claim 1, wherein theplurality of cameras are arranged in a non-grid format comprising: afirst set of cameras forming a first camera array; and a second set ofcameras forming a second camera array.
 4. The camera array of claim 1,wherein the non-grid format is selected from the group consisting of: acircular pattern, zigzagged pattern and scattered pattern.
 5. The cameraarray of claim 1, wherein at least two of the plurality of cameras areseparated by a distance approximating the separation of human eyes. 6.The camera array of claim 1, wherein the parallax confirmation andmeasurement module is further configured to measure parallax by ignoringpixels in the images captured by the plurality of cameras that are in anexposed occlusion set.
 7. The camera array of claim 1, wherein the imageprocessing module is further configured to estimate the size of an imageelement.
 8. The camera array of claim 1, wherein the image processingmodule is further configured to estimate the shape of a physical item.9. The camera array of claim 1, wherein the image processing pipeline isconfigured to select at least one distance as an “in best focus”distance and blur an image produced by the camera array based uponestimated distance information.
 10. The camera array of claim 9, whereinthe image produced by the camera array is an image captured by one ofthe plurality of cameras.
 11. The camera array of claim 9, wherein theimage processing pipeline module further comprises a super-resolutionprocessing module and the image produced by the camera array is a higherresolution super-resolved image synthesized using images captured by theplurality of cameras and parallax measurements from the parallaxconfirmation and measurement module to compensate for parallax in thecaptured images.
 12. The camera array of claim 1, wherein each cameraincludes a spectral filter configured to pass a specific spectral bandof light selected from the group consisting of a Bayer filter, one ormore Blue filters, one or more Green filters, one or more Red filters,one or more shifted spectral filters, one or more near-IR filters, andone or more hyper-spectral filters.
 13. The camera array of claim 1,wherein at least one camera is selected from the group consisting of: apolychromatic camera; and a near-IR camera.
 14. The camera array ofclaim 1, wherein the control circuitry of the cameras having differentimaging characteristics configure the cameras to operate with at leastone difference in operating parameters.
 15. The camera array of claim14, wherein the at least one difference in operating parameters includesat least one imaging parameter selected from the group consisting ofexposure time, gain, and black level offset.
 16. The camera array ofclaim 1, wherein the camera array is a monolithic camera array assemblycomprising: a lens element array forming the optics of each camera; anda single semiconductor substrate on which all of the pixels and controlcircuitry for each camera are formed.
 17. The camera array of claim 1,wherein the plurality of cameras are formed on separate semiconductorsubstrates.
 18. The camera array of claim 16, wherein the plurality ofcameras are mounted to a motherboard.
 19. A non-grid camera array,comprising: a plurality of cameras arranged in a non-grid formatconfigured to capture images of a scene, where each camera is formed ona separate semiconductor substrate and comprises: optics comprising atleast one lens element and at least one aperture; and a sensorcomprising a two dimensional array of pixels and control circuitry forcontrolling imaging parameters; and at least one spectral filter locatedwithin the camera, where each spectral filter is configured to pass aspecific spectral band of light; a controller configured to controloperation parameters of the plurality of cameras; and an imageprocessing pipeline module configured to process at least a subset ofimages captured by the plurality of cameras; wherein the plurality ofcameras comprises at least two cameras having different imagingcharacteristics including different resolutions; wherein the imageprocessing pipeline module comprises a parallax confirmation andmeasurement module configured to: measure parallax within the processedimages by: detecting parallax-induced changes that are consistent acrossthe images taking into account the position of the cameras that capturedthe images; and ignoring pixels in the images captured by the pluralityof cameras that are in an exposed occlusion set; generate a depth mapusing the measured parallax; and wherein the image processing pipelineis configured to select at least one distance as an “in best focus”distance and blur an image produced by the camera array based uponestimated distance information.
 20. A non-grid camera array, comprising:a plurality of cameras arranged in a non-grid format and configured tocapture images of a scene, where the plurality of cameras comprises amonolithic integrated module forming multiple cameras, where each cameracomprises: a lens element array forming the optics of each of theplurality of cameras, where the optics of each camera comprises at leastone lens element and at least one aperture; a single semiconductorsubstrate on which all of the pixels and control circuitry for eachcamera are formed; and at least one spectral filter located within eachcamera, where each spectral filter is configured to pass a specificspectral band of light; a controller configured to control operationparameters of the plurality of cameras; and an image processing pipelinemodule configured to process at least a subset of images captured by theplurality of cameras; wherein the plurality of cameras comprises atleast two cameras having different imaging characteristics includingdifferent resolutions; wherein the image processing pipeline modulecomprises a parallax confirmation and measurement module configured to:measure parallax within the processed images by: detectingparallax-induced changes that are consistent across the images takinginto account the position of the cameras that captured the images; andignoring pixels in the images captured by the plurality of cameras thatare in an exposed occlusion set; generate a depth map using the measuredparallax; and wherein the image processing pipeline is configured toselect at least one distance as an “in best focus” distance and blur animage produced by the camera array based upon estimated distanceinformation.